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Abstract 

Employing a randomized control group design, this study compared the efficacy of staff training 
using theory training, hands-on supervision and a self-instructional teaching manual. Participants 
were 12 undergraduate, health or social-work students. Initially, participants were given a three- 
hour lecture, and as a result, staff-participants increased their use of correct teaching procedures, 
but not to mastery. Participants in both groups continued to increase their teaching skills after 
intervention with either hands-on training or the self-instructional manual, though best effect was 
achieved with hands-on supervision. This effect was maintained at a two-month follow-up. At the 
end of the experiment, participants in the treatment manual group received hands-on supervision. 
As a result, their performances increased to the same levels as the participants in the hands-on 
supervision group. Hence, hands-on supervision was superior in increasing teaching performances. 
Participants receiving hands-on supervision reported that they were comfortable receiving this 
type of supervision, that the hands-on supervision was the best way to learn correct teaching 
procedures and that they would prefer hands-on supervision in a future job situation. Also, 
independent assessors rated hands-on supervision as the most suitable intervention. 

Keywords: Staff training, Supervision, Treatment manual, Hands-on supervision 


Introduction 

Developing effective staff training procedures to establish strong clinical skills in staff 
working with individuals with developmental disorders is important, since the clients' 
progress and well-being depends largely on the quality of the interventions they receive. 
Over the past four decades or so, a number of staff training procedures have been 
developed, including lecturing, treatment manuals, role-play, hands-on supervision, and 
computer simulation training (Arco, 2008; Downs & Downs, 2012; Eldevik, Ondire, 
Hughes, Grindle, Randell & Remington, 2013; Jahr, 1998; Thomson, Martin, Arnal, Fazzio & 
Yu, 2009; Thomson, Martin, Fazzio, Salem, Young, & Yu, 2012], Several studies have 
examined effects of lecturing and role play on staff competency. In an early study, Gardner 
(1972] used a randomized control group design to assess staffs’ clinical and theoretical 
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skills in working with individuals with intellectual disabilities. Results showed that theory 
training increased theoretical competency, but not clinical skills. Role play, in contrast, 
was effective in increasing clinical skills, but not theoretical skills. Similar results were 
obtained by Morch and Eikeseth (1992], who found that improved theoretical skills did 
not improve clinical skills, nor did it affect how the clients performed specific tasks under 
the supervision of the staff. 

Hands-on supervision (also described as on-the job training and direct feedback] is 
another procedure used to establish clinical skills, and has been demonstrated effective in 
several studies (Christian & Hannah, 1983; Fleming & Sulzer-Azaroff, 1989; Ivancic, Reid, 
Iwata, Faw, & Page, 1981; Iwata et al., 2000], Hands-on supervision involves direct 
feedback on teaching performances, and is similar to role playing with the exception that 
in hands-on supervision the staff is working directly with clients. 

Smith, Parker, Taubman, and Lovaas (1992] combined all three elements as they over the 
course of a one-week workshop provided staff working with individuals with intellectual 
disabilities theory training, role play, and hands-on supervision. By the end of the 
workshop, the staff participants showed improved theoretical and clinical skills, but when 
the participants subsequently were assessed on the same measures at their work place, 
skills acquired and demonstrated during the workshop did not generalize. 

Arco and du Toit (2006] examined effects of conventional group workshops and hands-on 
supervision in four staff participants working in a nursing home with a resident with 
problem behaviors. Results showed that after workshop, staff performance improved, but 
only one staff participant demonstrated competency. After hands-on training, all staff 
participants achieved and maintained competency. 

The use of treatment manuals is another way to improve staff competence. Arnal, Fazzio, 
Martin, Yu, Keilback, and Starke (2007] examined whether a self-instructional treatment 
manual detailing basic principles of applied behavior analysis and how to perform correct 
discrete trial teaching of specific targets, improved teaching competence. Results showed 
that all four participants’ discrete-trial-teaching improved as a result of the treatment 
manual, but only one of the participants achieved mastery of correct teaching skills, and 
for this participant, mastery was achieved for only one of three target behaviors. Using the 
same treatment manual, Fazzio, Martin, Arnal, and Yu (2009] added hands-on supervision 
in an attempt to further improve staff performances. Participants were five 
undergraduate psychology students. During baseline, participants were given 10 minutes 
to read a one-page program description of how to teach picture identification to a person 
acting in the role of an individual with autism. Next, participants studied the treatment 
manual, and after they could answer questions correctly probing the content of the 
treatment manual, their discrete-trial-teaching-skills were reassessed. In the final phase of 
the study, hands-on supervision was provided to all participants. Results showed that 
correct discrete-trial-teaching-skills increased for all participants over the course of the 
study, but to achieve mastery of correct discrete-trial-teaching, all participants required 
some degree of hands-on supervision. For four of the participants, only one session of 
hands-on supervision was required. The other participant required three sessions of 
hands-on supervision to achieve mastery. Subsequent tests assessing generalization of 
skills to new programs and to training of children with autism showed an increase in 
correct discrete trial teaching. 

The present study was designed to compare the efficacy of hands-on supervision to the 
efficacy of using a self-instructional treatment manual to increase staff competency. 
Initially, all participants were given three hours of lecture covering basic principles of 
applied behavior analysis and key elements of the training procedure. The lecture was 
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given at the start of the intervention because it is common in clinical practice to either 
provide theoretical training in some form or make sure that the staff possess sufficient 
theoretical competency before they start clinical work. At the end of the experiment, the 
group who received the least effective intervention was trained with the intervention 
demonstrated to produce the greatest effect. Two areas of performance were measured: 
(a] theoretical skills and (b] clinical skills. Staff competencies were assessed pre 
intervention, after theory training, after either hands-on supervision (Group 1] or self- 
instructional manual training (Group 2], and finally, after the group who received the least 
effective intervention was trained with the most effective intervention. 

Method 

Participants and Client 

Participants were 12 health or social-work bachelor's students (10 women and 2 men] 
between 21 and 35 years. They had no prior training in behavior analysis and discrete trial 
teaching. Participants were labeled ‘staff participants', to avoid confusing the participants 
with the client. 

The client was a 22 year old male with autism, intellectual disability and cerebral palsy. He 
used an activity schedule to perform simple tasks. He could read a number of two-to- 
three-word sentences, and answer simple questions. He followed simple instructions and 
performed various independent living tasks with varying degrees of staff assistance. 

Setting, Materials and Target Behaviors 

The client participated in a respite program for individuals with intellectual disabilities, 
and the study was carried out in that unit. A Panasonic 3CCD digital video camera was 
used to record all sessions. 

Over course of the experiment, the client performed 12 target behaviors, one for each of 
the 12 staff participants. The target behaviors were as follows; sorting towels, starting a 
laundry machine, starting a dryer, making instant hot chocolate, heating porridge, making 
instant soup, making a toast, emptying a trashcan, sorting trash for recycling, heating 
pizza, boiling eggs, and shredding paper and refilling paper in a Xerox machine. These 12 
target behaviors consisted of 10 responses each, and except for two target behaviors 
(sorting towels and sorting trash for recycling], an activity schedule was used to establish 
the target behaviors. The activity schedule consisted of an A5 binder with photos (4 x 5 cm 
with neutral background] of each of the 10 responses to be performed. 

Each staff participant was randomly assigned 1 of thel2 target behaviors. The target 
behaviors were selected because they were judged to be of similar degree of difficulties for 
the staff participants to teach and for the client to perform. The client had no previous 
experience with any of the target behaviors. 

Dependent Variables 

Dependent variables were (a] theoretical competence in behavior analysis, and (b] correct 
teaching skills. Video recordings of teaching sessions were scored by the first author and a 
confederate after the sessions. The multiple-choice test was scored by the first author. 

Theoretical Competency in Applied Behavior Analysis. All staff participants were tested 
for theoretical competence in behavior analysis, using a 21-item multiple choice test. The 
test assessed basic principles of behavior analysis, specifically focusing on reinforcement, 
stimulus control and prompt and prompt fading. 

Teaching Skills. Correct teaching skills were operationally defined as follows: (1] The staff 
participant let the client choose a putative reinforcer prior to starting the task, and 
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delivered the reinforcer to the client within three seconds of completing the task. (2] After 
the client had chosen a reinforcer, the staff participant presented a clear and concise 
verbal instruction to initiate the task. (3] The staff participant praised each correct 
response using language and intonation that varied across responses. (4] Whenever the 
client performed an incorrect response, or stopped responding for five seconds, the staff 
participant prompted the correct response; whenever the client stopped responding for 
five seconds, the staff participant prompted the next response in the chain. A correct 
prompt was defined as using manual guidance to initiate the correct response while 
gradually removing the prompt once the client started responding. Whenever prompt was 
used, reconstruction and positive practice was used until the client performed that specific 
response correctly without prompt. 

A total of 12 points could be obtained. One point was given if the staff participant let the 
client choose a putative reinforcer and delivered it within three seconds of completing the 
task. Another point was given if the staff participant presented a clear and concise verbal 
instruction to initiate the task. One point was given for each of the 10 responses defining 
the target behavior if praise was given upon a correct response, or if prompt, 
reconstruction and positive practice was performed correctly for incorrect responses or 
after 5 second of no responses. 

Independent Variables 

Theory training (lecture]. The theory training supervision lasted for three hours, covering 
basic principles of behavior analysis, specifically focusing on reinforcement, discriminative 
stimuli and prompt and prompt fading. The lecture covered all topics assessed in the 
multiple-choice-test, and it described, in detail, the procedures for teaching the particular 
target behavior each participant was to teach the client. Specifically, this included how to 
use the activity schedule, the correct use of reinforcement, correct presentation of SDs, 
procedures for prompt and prompt fading, and how to use reconstruction and positive 
practice after incorrect responses. Participants could take notes and ask questions during 
the lecture, but they were no handouts or videos. The class was given by the first author 
individually to each of the staff participant. 

Self-instructional manual 

The self-instructional manual consisted of four pages detailing (1] behavioral objective 
and justification for choice of the target behavior, [2] training materials, [3] the particular 
verbal instruction to start the training session, [4] description of each of the target 
behaviors, [5] procedure for prompting, reconstruction and positive practice in case of 
incorrect and no-responses, and [6] a procedure for addressing aggressive or self- 
injurious behavior, to be implemented if needed during the training session. The training 
session lasted for a maximum of 2 hours. 

Hands-on supervision 

During hands-on supervision, direct feedback on participants' teaching behaviors was 
provided by the supervisor. Behavior specific praise was given to the staff participant 
when he/she performed a correct teaching behavior. When the staff therapist performed 
incorrectly, he/she received immediate feedback, including a detailed description of the 
correct teaching of that behavior. If the participant performed incorrectly on the same 
behavior for a second time, the experimenter modeled the correct teaching behavior, and 
modeling was used until the participant performed the teaching behavior correctly. 
Hands-on supervision for all staff participants was provided by the first author. 
Participants received no written instructions. The training session lasted for a maximum 
of 2 hours. 
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Experimental Design 

A randomized group design was used to compare the effects of the self-instructional 
manual and the hands-on supervision. Participants were randomly assigned to either (a] 
the self-instructional manual group or (b] to the hands-on supervision group. After 
training (with hands-on supervision or the self-instructional manual], the group who 
received the least effective intervention was subsequently trained with the most effective 
intervention. All participants in both groups were given the theory course before receiving 
hands-on supervision or the self-instructional manual. This was done to assess the extent 
to which theory training alone would affect staff performances. 

Procedure 

Phase 1. Pretest. Prior to training, all participants in both groups were assessed on 
theoretical competence (i.e., the multiple choice test] and correct teaching skills (with the 
client present]. 

Phase 2. Theory training and retest. After the pretest, all participants in both groups were 
given the theory training. Because all staff participants were assigned a different target 
behavior to teach, the theory training was provided individually to each staff participant. 
Immediately thereafter, theoretical competence and teaching skills were reassessed. 

Phase 3. The self-instructional manual and hands-on training. Participants in both groups 
received four training sessions. At the end of each training session, participants in both 
groups were assessed on teaching skills. 

Self-instructional manual. Participants in this group were provided with treatment 
manuals for the following programs; sorting towels (Participant 1], starting a dryer 
(Participant 2], making instant hot chocolate (Participant 3], heating porridge (Participant 
4], making instant soup (Participant 5], and making a toast (Participant 6], The 
participants studied the manual for 90 minutes and answered a seven-item multiple- 
choice-test about its contents. A 100 % correct score was required for the session to end. 
Participants then conducted four consecutive training sessions. Between each training 
session, participants had a 10-minute break during which they could re-examine the self- 
instructional manual. 

Hands-on supervision. Participants in this group received hands-on supervision while they 
were conducting the four training sessions. The target behaviors were as follows; starting 
laundry machine (Participant 7], emptying trashcans (Participant 8], sorting trash for 
recycling (Participant 9], heating a pizza (Participant 10], boiling eggs (Participant 11], 
and shredding paper and refilling a Xerox machine (Participant 12]. 

Phase 4. Follow-up. Participants were retested on teaching skills one and two months after 
completion of the four training sessions. 

Phase 5. Training with the Most Effective Intervention. After follow up, the treatment 
manual group received hands-on supervision, as this turned out to be the more effective 
intervention. Hands-on supervision was identical to the one given to the hands-on group, 
except that the first author or her confederate acted in the role of the client. Then, the 
participants conducted four training sessions with the actual client. 

Control for effect of repeated measures on test of theoretical competence 

The effects of repeated measures on theoretical competence was examined by having 10 
second-year students from a bachelors program in social work take the same multiple- 
choice test on consecutive days with no prior instructions. 
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Reliability 

Inter observer reliability data were collected for all participants across all test and training 
sessions of this study. The observers were the first author and a colleague from the 
university. Reliability data were scored from video recordings. Reliability was calculated 
using point-by point agreement, with total number of recordings with agreement divided 
by total number of recordings times 100. The mean inter observer reliability for the 
teaching skills 98.7 (range 90 to 100 %]. 

Treatment Integrity 

Treatment integrity data was collected on all participants, assessing the extent to which 
the theory course, the hands-on supervision and treatment manual condition was carried 
out correctly according to the treatment protocol. Treatment integrity was scored from 
video recordings by the first author and a colleague. Results showed that the theory 
course was conducted 100 % correctly according to the protocol. In the manual group, 100 
% of the participants used the assigned time to study their manuals before and between 
the training sessions, and 100 % of the participants obtained a perfect score on the 
multiple-choice test. For the hands-on supervision group, the mean correct hands-on 
supervision during the four training sessions was 95% (range 93% - 97%]. 

Social validity 

To obtain an independent evaluation of social validity, six health bachelors level student 
who were not informed about the purpose of the study and who had no prior knowledge 
of behavior analysis were shown video clips from the training. The video clips showed 30 
minutes of the training from session four of Phase 3. Each 30 minute clip consisted of 3, 5- 
minute segments showing 3 randomly selected participants in the manual group, and 3 
randomly selected participants in the hands-on training group. Immediately after viewing 
the video, they were asked to rate which participants appeared more comfortable during 
the training sessions, which type of training seemed more suitable, and finally, which type 
of training they would prefer to receive themselves. 

Social validity was also evaluated by assessing the participants' satisfaction with the 
training. At the end of the training, participants completed a questionnaire containing 10 
statements concerning the aims of the study, the training procedure used, and the results 
that were achieved (see Table 2], Statements were scored on a five-point Likert scale 
going from "Strongly Disagree" (1 point] to "Strongly Agree" (5 points]. 

Results 

Figure 1 and Table 1 show mean correct teaching skills for the staff participants in both 
groups. As can be seen, the mean correct teaching skills during the pretest for participants 
in both groups was three percentage. After theory training, participants in both groups 
improved their teaching skills significantly (£(5] = 2.696, p = .043; t(5] = 3.381, p = .020, 
respectively for the teaching manual group and the hands-on group]. Mean correct 
teaching performance was 25 % for the self-instructional manual group and 56 % for the 
hands-on group, a nonsignificant between-group difference. After training with hands-on 
supervision or the self-instructional teaching manual, mean teaching skills for the hands- 
on group was 81 % correct, and the mean teaching skills for the self-instructional manual 
group was 50 % correct, a statistically significant group difference. At the two-month 
follow-up, mean correct teaching skills for the hands-on group was 87 %, and the mean 
correct teaching skills for the self-instructional manual group was 50 %, also a statistically 
significant group difference. When the participants in the training manual group received 
hands-on supervision, mean correct teaching skills reached 90 %. 
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Table 1. Mean percentage 
significance level for hands-on 

correct teaching skills, standard 
group and training manual-group. 

deviation, 

t-score and 

Variable 

Self-instructional Manual 

Hands-on supervision 

t-test 


Mean 

SD 

Mean 

SD 

t(10] 

Pretest 

3 

4.3 

3 

4.3 

0.00 

Post theory 
class 

25 

22.9 

56 

33.6 

2.77 

Post Training 

50 

25.2 

81 

15.5 

6.38* 

Follow up 

50 

25.0 

87 

9.5 

9.38* 


* p < .05. 
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Figure 1. Percentage correct teaching performances for the hands-on supervision group and 

the self-instructional teaching manual group across the different study phases 

The mean correct pretest score on the multiple-choice test of theoretical competence was 
51 % for the manual group and 52 % for the hands-on group. After completing the theory 
training, the mean correct score was 74 % for both groups. After intervention with hands- 
on supervision or the self-instructional manual, mean correct answer was 79 % for the 
training manual group and 83% for the hands-on group. None of the above group 
differences were statistically significant. 

To assess the possibility of test re-test reactivity, 10 participants (undergraduate health 
and social care students] were given the multiple-choice test on three consecutive 
occasions without receiving any training. Results showed that mean correct score was 38 
% on the first test; 39% on the second, and 40 % on the third test. 

The social validity questions answered by the participants and the mean results are shown 
in Table 2. Eleven out of 12 participants completed the form. As can be seen, participants 
receiving hands-on supervision reported that they were comfortable receiving hands-on 
supervision (mean score of 4.8 out of 5], The treatment manual group, who received both 
interventions, stated that the hands-on supervision was the best way to learn correct 
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teaching (mean = 4.9], All participants stated that they would prefer hands-on supervision 
in a future job situation (mean = 4.6], 

Table 2. The Social Validity Questions and the Mean Score for Each Question (a score of 1 
represents "Strongly Disagree" and a score of 5 represents "Strongly Agree") 


Social Validity Questions 


Score 


1. Was the information regarding the study presented in a satisfactorily way? 4.2 

2. Was the theory training sufficient to prepare for the subsequent training? 3.7 

3. Was the program manual easy to follow? 

(Scored by the training manual group only) 4.0 

4. Were you comfortable while receiving hands-on supervision? 

(Scored by the hands-on supervision group only) 4.8 


5. Was the hands-on supervision the best way to learn correct training? 


(Scored by the training manual group who subsequently received 

hands-on training) 4.9 

6. Were you satisfied with your own results? 4.1 

7. Would you prefer written feedback on your teaching performances? 2.1 

8. Would you prefer hands-on supervision in a future job situation? 4.6 


Four out of six independent panel members stated that the staff participants receiving 
hands-on supervision appeared more comfortable in the teaching situation. Two stated 
that the self-instructional manual staff participants seemed more comfortable. Three 
members stated that the client seemed more comfortable during the hands-on supervised 
training whereas two preferred the self-instructional manual sessions while one was 
neutral. Four panel member judged hands-on supervision as the most suitable condition 
(two were neutral), and five out of six stated that they themselves would prefer to receive 
hands-on supervision rather than the training manual. 

Discussion 

Employing a randomized control group design, this study compared the efficacy of staff 
training using theory training, hands-on supervision and a self-instructional teaching 
manual. Participants were 12 undergraduate, health or social-work students who had no 
prior knowledge of behavior analysis. Before training with either hands-on supervision or 
the self-instructional teaching manual, participants in both groups were given a three- 
hour lecture (i.e., theory training), covering key principles of behavior analysis and the key 
elements of the teaching procedure used for the particular client. As a result, staff- 
participants increased their use of correct teaching procedures, but not to mastery levels. 
After intervention with either hands-on training or the self-instructional manual, staff 
participants in both groups increased their teaching skills above the level they had 
reached after the theory training. However, the hands-on supervision group showed 
better teaching skills as compared to the participants who had received the self- 
instructional treatment manual. This effect was maintained at follow-up, conducted two 
months after completion of the training. 
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At the end of the experiment, participants in the treatment manual group received hands- 
on supervision. As a result, their performances increased to the same levels as the 
participants in the hands-on supervision group. Hence, hands-on supervision was superior 
in increasing teaching performances. 

Participants receiving hands-on supervision reported that they were comfortable 
receiving hands-on supervision, that the hands-on supervision was the best way to learn 
correct teaching procedures and that they would prefer hands-on supervision in a future 
job situation. Also, independent assessors who were not informed about the purpose of 
the study and with no knowledge of behavior analysis rated hands-on supervision as the 
most suitable intervention. Hence, the social validity of the hands-on supervision appears 
high, and the participants appeared satisfied and comfortable with the hands-on 
supervision they received. 

Results from the present study show, in contrast to previous studies (Gardner, 1972; 
Morch & Eikeseth, 1992], that the theory training may produce statistically significant 
improvements in the use of correct teaching procedures. This may be related to the fact 
that the theory training explained how to specifically teach the target behaviors involved. 
However, mastery criterion on clinical skills was reached only when theory training was 
followed by hands-on supervision. 

Though previous research suggest the theory training is not sufficient to establish 
competent staff skills (Gardner, 1972; Morch & Eikeseth, 1992], it cannot be determined 
empirically from the present study what effects, if any, repeated sessions of theory 
training would have had on staff performances, and this is a shortcoming of the present 
study. One way of examining this would be to include a third group that was given the 
initial theory training, but instead of receiving hands-on supervision or the treatment 
manual, were given additional theory training for the same length of time as the other 
groups received their intervention. 

Interestingly, theoretical competences improved slightly during the training with the self- 
instructional treatment manual and the hands-on training. This improvement could be 
attributed to test re-test reactivity, that is, that participants improved their test scores as a 
result of repeated testing. However, test-retest reactivity was assessed for a group of 12 
participants who only repeated the test (without receiving theory training or hands-on 
supervision or the treatment manual] and no significant improvements in scores was 
observed for this group. Hence, both hands-on supervision and the treatment manual 
seem to have improved the staff participants' theoretical competency. 

In the present study, all participants in both groups worked with the same client. This was 
done to make the teaching conditions across participants as similar as possible, avoiding 
the possible confound that some clients could be more difficult to teach than others. 
However, if all participants would teach the same skill to the same client, practice and/or 
ceiling effects could have confounded the results. To avoid this, each participant taught the 
client a specific target skill. Each target skill was judged to be of similar difficulty for the 
participant to teach and for the client to perform, and each participant was randomly 
assigned one of the target skills. Also, the client had no previous experience with any of the 
target skills. For 10 out of 12 target skills, a pictorial activity schedule was used to 
occasion each response of the target skill. For two of the programs (sorting towels and 
sorting trash for recycling], a pictorial activity schedule was not used because the client 
was unable to perform all responses of the target behaviors using picture prompts. One of 
these targets belonged in the hands-on supervision group and the other in the treatment 
manual group. Also, there were no significant between-group differences at the pretest 
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and after the theory course. This suggests that the target programs were of similar 
difficulties. 

The present study did not assess generalization across clients and target skills, and this is 
a limitation of the study. Another limitation of the study is the low number of participants. 
Nevertheless, the present study replicates and extends findings obtained by previous 
investigators. Findings indicate that theoretical training resulted in an increased use of 
correct teaching procedures, but not to mastery levels. Adding on hands-on training or the 
use of a self-instructional teaching manual resulted in a further increase in teaching skills, 
though hands-on supervision was more effective that the self-instructional teaching 
manual. Participants receiving hands-on supervision reported that they were comfortable 
receiving hands-on supervision and that they would prefer hands-on supervision in a 
future job situation. Also, independent assessors rated hands-on supervision as the most 
suitable intervention. 
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